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Abstract 


This  paper  presents  observations  and  insights  from  experiment  senior  mentors,  warfighters,  and 
experiment  management  personnel  from  five  Joint  Expeditionary  Foree  Experiments  (JEFX). 
The  design,  planning,  execution,  and  assessment  of  a  large-scale  command  and  control 
experiment  are  addressed.  In  addition,  experiment  management  changes  for  JEFX  06  are 
presented. 


USAF  Joint  Expeditionary  Force  Experiments 
Experiment  Management  Lessons  Learned 


Background 

Joint  Expeditionary  Force  Experiment  is  a  large-scale  Air  Force  experiment  designed  to  assist 
the  U.S.  Air  Force  in  preparing  for  the  challenges  of  the  2E'  Century  Expeditionary  Air  and 
Space  Force  operations.  To  that  end,  the  experiment  is  an  operational  innovation  activity  that 
attempts  to  anticipate  and  model  a  future  command  and  control  (C2)  system.  JEFX  04  was  the 
fifth  in  the  series  that  began  in  1998. 

JEFX  consists  of  three  spiral  events  followed  by  main  execution  with  approximately  5,000 
warfighters,  engineers,  analysts,  communicators,  and  public  affairs  participants.  Experiment 
concepts  and  initiatives  are  selected  based  on  their  potential  to  solve  Air  Force  capability 
shortfalls.  JEFX  04  included  sites  across  the  United  States  as  shown  in  Figure  1.  The  majority  of 
player  activities,  including  Live-fly,  are  performed  at  Nellis  AFB,  NV.  Modeling  and  Simulation 
support  is  provided  from  Hurlburt  AFB,  FL. 
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Figure  1 — JEFX  04  Participating  Sites 


During  each  experiment,  data  is  collected  from  experiment  participants  on  ways  to  improve  the 
management  of  the  experiment  itself  Observations  and  surveys  are  collected  via  a  web-based 
tool  from  senior  mentors,  players,  assessors,  and  support  personnel  during  each  experiment  on 
ways  to  improve  the  design,  planning,  execution,  and  assessment  of  JEFX. 

Design 

The  guidance  and  support  of  Air  Force  senior  leadership  is  instrumental  to  the  success  of  JEFX. 
The  Chief  of  Staff  of  the  Air  Force  (CSAF)  has  traditionally  provided  the  guidance  needed  early 


in  the  design  phase  to  scope  the  experiment.  AF  senior-leader  funding  support  has  allowed 
designers  to  adequately  model  a  future  command  and  control  system  as  well  as  allowing 
successful  transition  of  initiatives.  JEFXs  have  shown  that  during  the  initial  design  stage,  CSAF 
guidance  must  be  sought  out  prior  to  any  major  design  decisions  such  as  the  exclusion  of 
coalition  participants. 

The  CSAF  has  directed  some  experiment  management  changes  over  the  years  such  as  the 
direction  to  focus  the  experiment  on  near-term  capabilities  with  a  goal  of  85  percent  of  initiatives 
being  near-term  (i.e.,  transitionable  6-24  months  following  the  experiment)  and  15  percent  of  the 
experiment  focused  on  long-term  concept  development.  CSAF  reduced  the  scope  of  the 
experiment  by  limiting  the  number  of  initiatives  to  10  or  fewer,  based  on  the  attempt  in  JEFX  00, 
to  execute  and  assess  45  initiatives.  Operating  with  so  many  variables  was  difficult  and  follow- 
on  transition  was  not  achieved  to  the  extent  desired. 

Developing  a  method  to  transition  successful  initiatives  before  the  experiment  begins  is  a  key 
lesson  learned.  JEFX  began  programming  10  to  20  percent  of  the  experimentation  budget  for 
transition  of  successful  initiatives  to  encourage  healthy  initiative  submission  as  well  as  to  speed 
the  fielding  of  capabilities  sooner  than  traditional  acquisition  approaches.  The  Air  Force 
Experimentation  Office  (AFEO)  learned  that  the  transition  funding  must  lead  to  long-term 
sustainment  funding.  The  initiative  selection  criteria  were  changed  to  require  that  each  initiative 
have  a  sponsor.  A  sponsor  was  defined  as  an  organization  with  Program  Objective  Memorandum 
(POM)  authority  that  would  commit  to  long-term  sustainment  funding  if  the  initiative  was 
recommended  for  transition.  Sponsors  also  became  responsible  for  developing  a  transition  plan 
following  the  experiment.  A  Transition  Branch  was  formed  at  AFEO  to  handle  the  transition  of 
successful  initiatives  coming  out  of  experimentation.  Figure  2  shows  the  metrics  the  Transition 
Branch  developed  to  report  the  status  of  initiative  transition  to  the  AF/Xl. 
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Figure  2 — JEFX  Initiative  Transition  Metrics 


The  Five-year  Experimentation  Campaign  Plan  has  been  beneficial,  ft  has  improved 
experimentation  by  defining  the  long-term  schedule  and  resource  requirements.  AFEO  has 


learned  that  large-scale  field  C2  experiments  require  20  months  to  plan  and  execute  and  that 
experiments  involving  coalition  participants  can  require  even  longer.  This  determination  has  lead 
to  a  two  year  cycle  for  JEFX  experiments.  Figure  3  shows  the  draft  schedule  for  JEFX  06. 
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Figure  3 — JEFX  06  Draft  Schedule 


A  small-scale  (1  /20*  the  size  of  JEFX)  Advanced  Process  and  Technology  Experiments  (APTX) 
has  been  executed  twice  in  intervening  years  and  has  provided  a  more  continuous 
experimentation  methodology.  Currently,  APTX  Execution  is  envisioned  as  the  Spiral  0  for 
future  JEFX  events,  to  provide  better  linkage  between  experiments.  Previous  JEFX  lessons  are 
that  experiment  recommendations  must  be  available  in  time  for  immediate  POM  cycle 
submission  and  that  experiment  execution  must  be  deconflicted  from  the  summer  months.  These 
two  lessons  have  led  to  a  planned  experiment  execution  schedule  of  Oct  to  May  for  JEFX  06.  A 
three-spiral  schedule  followed  by  main  execution  with  a  minimum  of  6  weeks  between  spirals  to 
allows  time  for  innovation  to  occur  between  the  spiral  events. 

The  campaign  plan  has  been  more  effective  when  senior  leadership  has  directed  long-term, 
recurring  focus  on  specific  areas  of  interest  such  as  the  multi-experiment  Time  Sensitive 
Targeting  (TST)  effort  conducted  between  1999  and  2004,  which  resulted  in  tremendous  C2 
advances.  These  advances  were  used  in  Operation  IRAQI  FREEDOM.  Recent  JEFX  experiment 
results  have  drawn  attention  to  areas  such  as  Effects-Based  Operations,  which  will  require  a 
long-term  service  and  joint  effort — as  happened  with  TST — if  senior  leadership  decides 
improvement  is  needed.  Campaign  plans  will  be  also  be  useful  for  pre-identified  long-term 
efforts  such  as  Net-Centric  Warfare.  Past  lessons  indicate  a  5  to  8  year  period  is  needed  with 
both  service  and  joint  large-scale  field  experimentation  in  order  to  achieve  significant  results. 

Experimentation  campaign  plans  lead  to  experiment  results  that  influence  higher  level  capability- 
based  planning.  An  example  is  the  newly  developed  AF  Transformation  Flight  Plan,  which,  in 
turn,  influences  future  experimentation  campaign  plans.  JEFX  results  also  influence  AF 
capability-based  planning  by  providing  experiment  results  to  support  the  draft  Capability  Review 
and  Risk  Assessment  activities  and  CONORS  development  efforts,  as  shown  in  Figure  4. 


Figure  4 — JEFX  06  Potential  Linkages  to  Draft  AF  Capabilities  Based  Planning 


The  use  of  operational  threads  as  a  framework  for  the  design  of  an  experiment  has  been  useful. 
During  JEFX  04,  the  Army  Close  Air  Support  Situational  Awareness  (ACASSA)  initiative  began 
designing  operational  threads  for  their  experimental  processes.  In  parallel,  the  assessment  team 
began  identifying  operational  threads  as  a  way  to  demonstrate  operational  relevance  for  initiative 
capabilities.  A  thread  is  a  series  of  linked  tasks  that  provide  the  framework  for  measuring  an 
overall  capability  in  an  operationally  relevant  context.  This  logical  next  step  resulted  from 
previous  assessments  that  analyzed  TST  capability  by  assessing  the  more  measurable  Find,  Fix, 
Track,  Target,  Engage,  and  Assess  tasks.  It  was  learned  that  operational  threads  must  to  be 
designed  into  the  experiment  so  that  the  scenario  and  Master  Scenario  Events  List  (MSEL)  will 
support  the  assessment  and  so  that  the  threads  themselves  are  deconflicted  and  executed  as 
designed.  Using  operational  threads  as  a  design  technique  increased  the  use  of  architectural 
Operational  View  and  Technical  View  products  such  as  those  shown  in  Figure  5.  This  technique 
resulted  in  increased  rigor  of  the  experiment  design.  Detailed  “scripts”  were  built  for  the  threads 
that  documented  inputs,  controls,  systems,  activities,  and  outputs. 
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Figure  5 — JEFX  04  ACASSA  Operational  View 


Live  experimentation  events  or  “Live-Fly,”  in  JEFX  vernacular,  provide  the  end-to-end  test 
needed  to  assess  operational  utility.  Many  initiatives  require  Live-Fly  to  demonstrate 
capabilities.  Live-Fly  has  also  been  useful  to  validate  that  operational  level  processes  will 
support  tactical  level  execution.  Constructive  and  virtual  simulation  can  and  do  provide  the 
experiment  environment  at  a  lower  cost  and  are  used  to  train  participants  and  to  spiral-develop 
processes  and  systems.  It  is  difficult  to  innovate  when  Live-Fly  is  a  one-time,  short,  exercise-like 
event.  Therefore,  Live-Fly  duration  was  increased  from  5  to  9  days  in  JEFX  04  with  more  time 
to  innovate  between  missions.  It  was  also  learned  to  limit  participation  to  the  minimum  number 
of  live-fly  players  needed,  with  greater  participation  of  test  squadrons.  It  has  been  difficult  for 
experiment  designers  to  make  the  transition  from  the  constructive  phase  (supported  by  modeling 
and  simulation)  to  the  Live-Fly  phase  transparent  to  the  experiment  players.  A  10-mile  buffer 
around  the  ranges  has  been  used  to  separate  constructive  entities  and  live-fly  aircraft  on  the 
Common  Operational  Picture  for  safety-of-flight  reasons.  The  constructive  scenario  and  MSEL 
events  must  be  aligned  with  the  Nellis  range  configuration  and  events  to  improve  the  situation. 
Increased  use  of  virtual  platforms  during  the  constructive  phase  may  also  improve  the  transition. 

Combining  large-scale  field  experimentation  with  a  large  joint  exercise  is  not  a  complementary 
mix.  Conflicting  objectives  and  battle  rhythms  impede  innovation,  confuse  the  participants  and 
controllers,  and  can  result  in  negative  learning.  The  objective  of  an  experiment  is  to  learn  and 
discover,  but  an  exercise  is  designed  to  practice  what  already  is  known.  Warfighters  have  stated 
their  surprise  when  they  find  experimentation  to  be  much  more  difficult  than  participating  in  an 
exercise.  Warfighters  also  come  to  realize  experimentation  is  not  about  winning  the  war  or 
evaluating  their  performance.  Joint  participation  has  shown  to  be  very  beneficial  when  each 
Service’s  experiment  objectives  are  compatible.  Joint  operations  reveal  the  seams  between 
Services’  C2  and  leads  to  joint  and  service  Doctrine,  Organization,  Training,  Materiel, 
Leadership,  Personnel,  and  Facilities  (DOTMLPF)  recommendations  for  change. 

However,  combining  testing  and  experimentation  is  compatible  to  some  degree.  Smaller  scale 
testing  within  an  experiment  has  proven  very  easy  to  integrate  within  the  existing  control  and 


battle  rhythm  construct  of  an  experiment.  Early  spiral  events  are  focused  on  technical  assessment 
and  are  very  similar  to  tests.  The  live-fly  venue,  however,  has  many  uncontrolled  variables  and 
inadequate  instrumentation.  To  date,  it  has  not  provided  the  environment  needed  to  conduct 
rigorous  testing. 

The  initiative  selection  process  is  never  over.  Senior  leadership  can  override  early  decisions,  and 
late  initiatives  have  become  a  routine  part  of  every  experiment,  requiring  revisions  to 
architectures,  floor  plans,  and  other  elements  of  the  experiment.  The  lesson  learned  here  is  to  be 
flexible  and  build  in  room  for  growth  in  an  experiment.  This  room  for  growth  must  include 
budget,  schedule,  and  space.  There  is  a  point  of  diminishing  returns  however.  In  JEFX  this  point 
occurs  at  Spiral  2.  After  Spiral  2,  new  initiatives  are  not  usually  integrated  well  into  the 
experiment  and  often  end  up  as  stand-alone  demonstrations. 

Planning 

Augmenting  existing  organizations  with  permanent  experimentation  billets  has  shown  to  be  a 
valid  organizational  construct.  The  JEFX  organization  is  the  Experimentation  Enterprise.  This 
enterprise  consists  of  existing  AF  organizations  that  have  been  augmented  with  permanent  staff 
to  work  JEFX  tasks.  The  JEFX  Experiment  Enterprise  consists  of  Electronic  Systems  Command 
at  Hanscom  AFB,  505*  Command  and  Control  Wing  at  Hurlburt  AFB,  and  the  AFC2ISRC  and 
ACC/SC  at  Langley  AFB.  Augmenting  existing  organizations  rather  than  forming  a  large 
experimentation  organization  allows  significant  organizational  flexibility.  Experimentation  work 
is  often  shifted  to  other  staff  when  required.  The  organizations  with  permanent  staff  funded  by 
experimentation  have  become  known  as  the  “4-ship.”  The  Air  Force  Experimentation  Office,  as 
part  of  the  AFC2ISRC,  is  the  flight  lead,  and  the  other  members  have  distinct  roles  and 
responsibilities.  The  4-ship  provides  JEFX  an  in-place  collaborative  team  that  crosses  many 
“tribal”  lines  and  provides  JEFX  the  ability  to  quickly  tap  into  a  wealth  of  expertise  to  get  things 
done. 

A  method  is  needed  to  document,  update,  and  track  manning  by  function,  software,  and  hardware 
requirements  and  configuration.  An  Operator,  Position,  System,  Application,  Training  and 
Hardware  (OPSATH)  document  has  been  used  to  plan  hardware  and  software  configurations  for 
individual  participants  and  to  help  in  the  development  of  individual  profiles  and  passwords.  The 
OPSATH  is  a  Microsoft  Excel™  spreadsheet  populated  with  all  experiment  manning  positions 
and  all  the  software  options.  This  information  has  been  shared  via  periodic  publishing,  but  due  to 
the  dynamic  reconfiguration  required  during  an  experiment,  it  has  always  served  as  a  planning 
document  rather  than  a  configuration  control  document.  It  has  been  proposed  that  the  Excel 
spreadsheet  become  part  of  an  online  database  on  the  JEFXLINK  Website  so  that  it  can  be 
accessed  real-time  from  any  location  and  thereby  usable  as  a  configuration  management  tool 
during  execution. 

A  web  portal  is  essential  to  share  and  manage  information  across  the  experiment  enterprise 
efficiently.  The  JEFXLINK  Website  shown  in  Figure  6  has  been  instrumental  to  the  sharing  of 
planning  and  execution  information. 
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Figure  6— JEFXLINK  Website 


JEFXLINK  provides  a  virtual  workspace  to  immediately  share  information  across  this  enterprise. 
In  addition  “web  tools”  were  added  to  help  manage  manning  and  assessment  tasks  and  to 
encourage  collaboration.  A  small  team  of  AFEO-dedicated  web  developers  has  identified  unique 
experiment  requirements  and  added  functionality  over  a  6-year  period.  Having  dedicated  web- 
tool  developers  rather  than  a  “pick-up”  team  each  experiment  has  proven  to  be  a  key 
management  decision.  Event  registration,  participant  profile  information,  Deployment 
Requirements  Manning  Document  (DRMD)  development,  surveys,  web  forums,  web  chat, 
private  journals,  document  collaboration,  a  calendar,  and  a  problem  reporting  system  are  now 
being  accessed  online  on  NIPRNET,  SIPRNET  and — during  JEFX  04 — on  a  Coalition-NET. 
Additional  functionality  is  being  added  so  that  operational  threads  can  be  built  and  accessed 
online  and  so  that  DRMD  and  OPSATH  databases  are  built  online  with  linked  databases. 

The  integration  of  Coalition  members  into  an  experiment  requires  additional  effort,  planning 
time,  and  senior  leader  support.  Figures  7  and  8  show  the  DOD  Information  Technology  Security 
Certification  &  Accreditation  Process  (DITSCAP)  and  data  disclosure  steps  and  timelines 
executed  during  JEFX  04. 
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Because  of  their  need  to  support  ongoing  real  world  activities,  it  can  be  difficult  to  obtain  the 
support  of  policy  and  technical  groups  to  review  and  approve  experimentation  architectures  and 
to  develop  release  policies.  Experimentation  needs  senior  leader  support  or  experimentation  must 
be  elevated  in  importance  or  the  steps  in  this  process  cannot  be  completed  in  8  months,  as  was 
needed  for  JEFX  04.  In  JEFX  04,  AF  obtained  the  support  of  the  Chairmen  of  the  JCS  in  order  to 
achieve  the  experimentation  timeline.  Experimentation  architectures  cannot  be  developed  earlier 
than  8  months  ahead  of  time,  because  initiatives  and  supporting  infrastructure  are  being  defined 
and  engineering  innovation  is  occurring  as  the  experiment  design  is  being  developed. 

Execution 

A  professional  control  force  is  important  to  execution  success.  Their  subject  matter  expertise  on 
scenario.  Master  Scenario  Events  List  (MSEL)  and  modeling  and  simulation  makes  the 
experiment  run.  A  mutually  supportive  relationship  must  exist  between  assessment  and  control 
teams.  Each  team  depends  on  the  other  for  feedback  to  achieve  experiment  objectives.  The  use  of 
the  Joint  MSEL  program  by  assessors  and  controllers  to  collaborate  dynamically  was  shown  to 
be  a  very  useful  capability  in  JEFX  04.  The  program  must  be  modified,  however,  so  that  the 
taxonomy  of  an  experiment  is  supported;  for  example,  operational  threads  and  initiative 
capabilities  rather  than  training  objectives. 

Senior  mentor  participation  in  experimentation  promotes  sharing  lessons  and  resolving  issues. 
Most  senior  mentors  are  involved  with  other  joint  and  service  experiments  and  wargames.  Their 
participation  in  other  experimentation  venues  helps  share  lessons  across  the  joint  force  and 
promotes  resolving  problems  at  the  general-officer  level  if  needed.  The  execution  team,  the 


assessment  team,  and  the  Joint  Force  Air  Component  Commander  have  been  assigned  dedicated 
mentors  in  past  experiments  and  these  senior  mentors  have  been  proven  to  be  useful. 

The  spiral  development  and  assessment  process  shown  in  Figure  9  has  been  a  very  successful 
approach  to  developing  capabilities.  It  is  essentially  a  building-block  approach  to  construction  of 
a  future  C2  architecture.  A  minimum  of  6  weeks  is  needed  to  allow  innovation  to  occur  between 
spirals  and  to  allow  issues  to  be  identified  and  addressed  before  the  next  event.  The  assessment 
activities  become  more  focused  on  operational  utility  as  the  warfighters  become  trained  and 
proficient,  the  architecture  is  completely  connected  and  stabilized,  and  technical  assessment  is 
completed. 
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Figure  9 — Spiral  Development/Assessment 


A  permanent  infrastructure  increases  the  stability  and,  therefore,  the  ability  to  demonstrate 
capabilities  in  an  experiment.  The  building  of  the  Combined  Air  Operations  Center-Nellis 
(CAOC-N)  built  in  2003  and  shown  in  Figure  10  provided  a  permanent  infrastructure  to  JEFX  to 
maintain  the  architecture  between  spirals. 


Figure  10 — Combined  Air  Operations  Center-  Neliis 


In  previous  experiments,  trailers  were  used  and  systems  were  dismantled  between  spirals.  This 
approach  was  costly  and  impeded  the  development  of  a  stable  architecture.  It  is  envisioned  that 
placing  this  facility  at  a  tactical  training  base  can  provide  better  integration  of  the  operational 
level  (e.g.,  producing  strategy  and  Air  Tasking  Orders)  and  tactical  level  of  war  (e.g.,  creating 
effects  by  dropping  bombs). 

The  ability  to  maintain  configuration  control  of  the  architecture  while  encouraging  innovation  is 
a  challenge.  By  nature,  innovation  requires  daily  tinkering  with  systems  configuration.  Having  a 
stable  and  documented  configuration  requires  some  management  of  this  change  process.  Rules  of 
Engagement  must  be  established  for  contractors  and  initiative  providers  in  order  to  control 
changes.  The  challenge  is  to  allow  rapid,  dynamic  changes  while  still  maintaining  a  stable  and 
documented  architecture. 

Maintaining  player  team  continuity  during  an  experiment  is  required.  Weeks  of  training  is 
conducted  prior  to  main  execution  to  provide  the  participants  the  best  understanding  and  ability 
to  use  new  initiatives.  Having  players  leave  after  one  or  two  spirals  reduces  the  ability  of  the 
warfighters  as  a  group  to  execute  the  new  processes  with  new  systems.  Turnover  has  been 
minimized  in  experiments  by  soliciting  strong  senior  leadership  participation  early  in  the 
experiment  planning  stages  and  requiring  all  manning  changes  be  vetted  through  them  during 
execution.  Another  major  change  that  reduced  manning  turnover  was  the  establishment  of  a  lead 
Number  Air  Force  as  the  “core  NAF”  for  each  experiment,  rather  than  manning  the  experiment 
with  a  “rainbow  NAF”  approach.  The  establishment  of  a  dedicated  “experimentation  NAF”  is 
being  explored  with  8*  AF  returning  for  the  second  experiment  in  a  row.  The  hope  is  that  some 
corporate  knowledge  will  return  with  the  same  NAF.  Warfighters  recognized  the  differences 
between  experimentation  and  exercises  after  the  first  experiment  and  now  should  be  more 
prepared  to  achieve  experimentation  objectives  in  the  future. 

The  continuity  of  experiment  enterprise  manning  has  reduced  repeat  lessons  learned  and 
permitted  a  steady  development  of  experimentation  knowledge.  Experiment  designers,  engineers, 
controllers,  and  assessors  have  been  a  lead  group  of  civilian  contractors  who  have — over  a  series 
of  experiments — developed  a  body  of  knowledge  on  how  to  design  and  execute  large-scale  field 


C2  experiments.  The  experimentation  eommunity  has  had  a  healthy  turnover  but  has  also 
retained  many  contractors  from  experiment  to  experiment  who  have  built  up  the  corporate 
knowledge  level.  Corporate  knowledge  is  documented  on  JEFXLINK  Website  in  the  artifacts 
and  reports  of  each  experiment. 

Tension  occurs  between  “experimenting  to  learn”  and  “experimentation  to  field.”  JEFX  has 
always  been  part  disco  very/experiment,  part  hypothesis-testing,  and  part  demonstration.  The 
pendulum  in  recent  experiments  has  swung  more  toward  experimentation  to  field  and  is  expected 
in  the  future  to  move  beyond  just  demonstrating  to  testing  before  fielding.  The  commingling  of 
these  different  types  of  experiments  is  not  without  issues.  Initiatives  may  not  be  properly 
demonstrated  if  significant  infrastructure  innovation  is  occurring  during  discovery 
experimentation.  Balancing  these  different  types  of  experiments  has  required  temporal 
separation,  which  is  usually  accomplished  by  conducting  most  innovation  during  or  between 
spiral  events  and  reserving  the  main  execution  for  the  official  demonstration  phase.  This 
approach  has  worked  well  with  technical  assessments.  During  the  first  part  of  each  spiral,  the 
technical  assessment  activities  are  scheduled  as  separate  from  the  player  activities,  if  possible. 

But  in  every  experiment,  so  many  changes  and  activities  are  occurring  daily  that  the  saying  is,  “if 
you  think  you  know  whaf  s  going  on,  you  obviously  don’t  know  whaf  s  going  on.”  This  situation 
leads  many  participants  to  feel  constantly  off-balance  and  unprepared  for  execution;  not  knowing 
exactly  how  things  will  work  day  to  day. 


Assessment 

Determining  ground  truth  in  live  experiments  is  difficult.  Because  of  the  scope  and  dynamic 
nature  of  JEFX  experiments,  it  is  difficult  to  control  all  the  variables,  especially  in  live- fly 
activities.  The  live-fly  experiment  is  just  one  of  many  events  on  the  range  each  day  and  live-fly 
systems  are  not  all  instrumented  as  they  would  be  for  a  test  event.  A  balance  must  exist  here. 
Instrumentation  costs  money  and  buying  up  all  the  systems’  time  on  the  range  for  a  period  of 
time  can  be  cost-prohibitive.  However,  flexibility  is  needed  in  experimentation  because  play  can 
be  dynamic  and  the  need  for  data  on  one  particular  target  or  threat  is  not  always  predetermined. 
The  lesson  learned  here  is  to  reduce  the  time  and  range  area  needed  for  ground  truth  ahead  of 
time,  buy  only  what  is  needed,  and  articulate  not  only  what  must  happen  to  support  the 
experiment  but  also  what  activities  on  the  range  must  not  occur.  A  range  liaison  to  the 
assessment  team  would  help  avoid  many  of  these  problems. 

Warfighters  in  the  experiment  contribute  to  the  assessment  of  process  and  system  initiatives. 
They  are  not  evaluated  for  their  performance  but  are  in  the  experiment  to  provide  the  assessment 
team  a  warfighter’s  perspective  on  the  value  of  an  initiative — a  subjective  assessment  of  the 
initiative.  Objectives  measures  are  collected  by  the  assessment  team  to  augment  this  subjective 
assessment  as  well  as  objective  technical  assessment  of  initiative  performance.  It  usually  takes 
warfighters  some  time  before  they  realize  “winning  the  war”  is  not  the  focus  of  experimentation 
and  that  their  individual  performance  is  not  being  assessed  except  in  terms  of  their  expertise  to 
provide  valid  assessment. 

The  technique  of  determining  what  issues  must  be  addressed  in  the  final  report  first  and  then 
building  a  team  around  the  report  has  proven  to  be  a  successful  approach.  In  other  words,  build 
your  assessment  team  from  the  final  report  back.  The  Assessment  Plan  and  support  team  is 
unique  for  each  experiment.  Depending  on  the  focus  of  the  experiment,  operational  and  technical 
assessment  subject  matter  experts  are  chosen  to  assess  the  demonstration  of  capabilities.  These 
assessors  must  be  independent  of  the  initiative  and  must  have  the  understanding  of  both  the 


process  and  programming  that  this  capability  is  proposed  to  transition  into.  A  core  staff  that 
returns  to  each  experiment  and  that  is  augmented  by  experts  covering  DOTMLPF  categories  has 
been  a  useful  approach,  since  JEFX  reports  to  date  have  required  recommendations  for  these 
categories.  Addressing  senior  leadership  issues  and  questions  during  the  planning  phase  and 
updating  the  categories  of  the  report  as  senior  mentors  determine  new  issues  during  execution  is 
key  to  being  on  target  with  the  final  results. 

Automation  must  be  developed  to  quickly  determine  C2  process  results.  Because  C2  weapon 
systems  have  not  been  designed  with  an  intrinsic  debrief  capability,  the  manual  collection  of  data 
is  often  required.  Data  from  several  system  and  operator  locations  must  be  recorded  by  assessors 
and  analyzed  as  a  group  to  determine  actual  results.  This  analysis  is  a  very  manpower-intensive 
and  time-consuming  process.  Automatic  collection  of  data  across  several  systems  and  production 
of  measured  results  of  the  activities  are  needed  at  the  end  of  each  day.  Some  improvement  has 
been  made  with  the  use  of  the  Modular  Analysis  and  Test  Support  System  (MAnTSS)  shown  in 
Figure  1 1  in  JEFX  04  to  collect  real-time,  live-fly  data  and  compile  an  operational  thread  result. 
However,  only  one  or  two  threads  were  reconstructed  per  day.  Also,  MAnTSS  was  not 
connected  to  the  coalition  network  and  systems;  therefore  it  did  not  collect  all  of  the  required 
information.  Due  to  a  lack  of  information  and  time,  a  live- fly  debrief  was  never  available  for 
more  than  6  of  the  30  or  more  CAOC  taskings  that  occurred  in  the  average  4-hour  range  period. 
A  better  C2  debrief  capability  is  needed  for  JEFX  experimentation. 
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Figure  11 — MAnTSS  Data  Flows 


In  order  to  quickly  collect  the  data  for  30  or  more  CAOC  taskings,  an  automated  data  collection 
capability  is  needed  on  the  CAOC  network  so  that  data  collectors’  observed  results  and  system 
information  can  be  stored  in  one  event  database.  If  all  C2  tasking  information  is  logged  and  data 
can  be  analyzed  from  any  workstation  by  the  assessment  team,  then  automated  reports  can  be 
generated  to  provide  an  acceptable  80-percent  solution  of  results  before  debriefs  begin. 


Way  Ahead 

A  conscious  management  decision  has  been  made  to  take  increased  risk  in  JEFX  06,  based  the 
advice  in  Power  to  the  Edge  and  Campaigns  of  Experimentation:  Pathways  to  Innovation  and 
Transformation.  The  main  engine  of  the  Air  and  Spaee  Operations  Center  (AOC)  system-of- 
systems  will  be  upgraded  with  Theater  Battle  Operations  Netcentrie  Environment  (TBONE). 

This  revolutionary  ehange  will  affect  several  areas  of  DOTMLPF  for  the  AOC  and  is  intended  to 
improve  network-eentrie  attributes  of  the  AOC.  The  Network  Centric  Operations  Conceptual 
Framework  version  2. 0  may  provide  a  framework  for  development  of  measures  and  the  reporting 
of  results. 

The  experimentation  management  organizational  structure  has  also  ehanged  to  one  built  around 
Capability  Development  Teams  (CDT).  These  AFC2ISRC  lead  cross-matrixed  teams  will  build 
operational  threads  to  focus  experimentation  on  steps  designed  to  aehieve  eapability  goals.  CDTs 
will  use  the  operational  thread  construct  to  design  into  the  experiment  the  tasks,  supporting 
systems,  personnel,  and  MSELs  needed  to  demonstrate  a  capability.  The  JEFXLfNK  operational 
thread  web  tool  shown  in  Figure  12  is  being  used  to  document  the  threads  online  in  order  to 
standardize  the  thread  construets  as  well  as  to  provide  a  virtual  eollaboration  workspaee.  The 
assessment  team  plans  to  build  the  measures  into  the  tasks  so  that  results  can  be  populated  online 
to  provide  easy  roll-up  of  results. 
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Figure  12 — Web-Based  Operational  Thread  Development 


The  spiral  development  of  live-fly  activities  will  occur  by  conducting  Live  Fly  during  both 
Spiral  3  and  Main  Execution  to  permit  time  for  innovation  to  occur  after  Spiral  3  as  lessons  are 


learned  about  building,  eontrolling,  and  using  an  airborne  IP  network.  Past  JEFXs  have 
demonstrated  that  establishing  a  stable  Link- 16  environment  was  an  almost  insurmountable 
challenge  and  the  adoption  of  the  spiral  development  process  to  live-fly  architectures  was  long 
overdue. 

Distributed  Operations  has  been  a  focus  of  JEFX  in  the  past,  but  was  not  adequately  supported 
by  system  capabilities  and  communications.  Currently,  an  increased  focus  on  Force  Level  and 
Unit  Level  collaboration  using  TBONE  is  planned  for  JEFX  06.  It  is  proposed  that  four  Wing 
Operations  Centers  be  modeled  to  replicate  bomber,  fighter,  and  Navy  units.  The  desire  is  to 
reduce  the  unit  manning  in  the  AOC,  provide  units  more  lead  time  in  planning,  and  improve  the 
effectiveness  of  the  assessment  of  effects. 

Finally,  the  extent  to  which  testing  can  be  combined  with  experimentation  will  be  explored  in 
JEFX  06.  As  part  of  the  TBMCS  1.1.4  upgrade,  the  AOC  will  undergo  Developmental  Test  and 
Evaluation  during  the  spiral  events  and  Operational  Test  and  Evaluation  during  main  execution. 
The  requirements  of  the  test  community  will  force  management  to  segregate  experimentation 
activities  in  time  and/or  space  to  eliminate  experimentation  variables  from  the  testing 
environment.  This  new  way  of  doing  business  brings  the  acquisition  community  in  early  to 
develop  test  and  support  plans.  This  change  should  improve  the  transition  of  successful 
initiatives  after  the  experiment. 

Summary 

The  management  of  a  large-scale  C2  experiment  such  as  JEFX  continues  to  evolve,  based  on 
both  internal  lessons  learned  and  lessons  learned  from  other  experiments  and  reports.  The  lessons 
on  design,  planning,  and  assessment  of  JEFX  have  been  captured  for  all  five  JEFX  experiments 
beginning  in  1998.  The  JEFX  experimentation  management  lessons  documented  here  contribute 
to  the  larger  body  of  knowledge  on  experimentation. 
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